Improving Classification Accuracy Assessments with Statistical Bootstrap Resampling Techniques

The use of remotely sensed imagery to generate land cover models is common today. Validation of these models typically involves the use of an independent set of ground-truth data which are used to calculate an error matrix resulting in estimates of omission, commission, and overall error. However, each estimate of error contains a degree of uncertainty itself due to 1) conceptual bias, 2) location/registration and coregistration errors, and 3) variability in the sample sites used to produce and validate the model. In this study, focus was not placed upon describing land cover mapping techniques, but rather the application of bootstrap resampling to improve the characterization of classification error, demonstrate a method to determine uncertainty from sample site variability, and calculate confidence limits using statistical bootstrap resampling of 500 sample sites acquired within a single Landsat 5 TM image. The sample sites represented one of five land cover categories (water, roads, lava, irrigated agriculture, and rangelands) with each category containing 100 samples. The sample set was then iteratively resampled (n=200) and 65 sites were randomly selected (without replacement) for use as classification training sites while the balance (n=35) were used for validation. Imagery was subsequently classified using a maximum likelihood technique and the model validated using a standard error matrix. This classification-validation process was repeated 200 times. Confidence intervals were then calculated using the resulting omission and commission errors. Results from this experiment indicate that bootstrap resampling is an effective method to characterize classification uncertainty and determine the effect of sample bias.